Prevent malformed OpenSearch log entries from crashing task log fetch#69307
Conversation
A single stored log entry with a non-string `event` field (for example, a task that logs a list or dict as the sole message argument) currently crashes the entire task-log-fetch request with an unhandled pydantic.ValidationError, instead of degrading gracefully. _read() built StructuredLogMessage objects from stored OpenSearch hits without catching validation failures. This catches ValidationError per hit and falls back to a stringified event, matching the existing fallback pattern in file_task_handler.py's _log_stream_to_parsed_log_stream. Related: apache#69304
|
Could the fallback avoid reusing the original Drafted-by: Codex (GPT-5); reviewed by @Vamsi-klu before posting |
potiuk
added
the
ready for maintainer review
_safe_build_structured_log_message runs once per stored hit inside the _read comprehension, so logging the fallback at warning level fires once per malformed line within a single log-fetch request. Drop it to debug so a request with many malformed entries does not flood the logs.
|
@Vamsi-klu Good catch on the code path. I left the timestamp as-is on purpose here: in both handlers the timestamp is not user data. It is written by |
The shared ti fixture defaults to RUNNING, and _read() now delegates running/deferred tasks to the base handler's remote-log path, which is not wired up in unit tests. Match the sibling read tests by setting the task to SUCCESS so the test exercises the stored-log fallback path.
…apache#69307) * Prevent malformed OpenSearch log entries from crashing task log fetch A single stored log entry with a non-string `event` field (for example, a task that logs a list or dict as the sole message argument) currently crashes the entire task-log-fetch request with an unhandled pydantic.ValidationError, instead of degrading gracefully. _read() built StructuredLogMessage objects from stored OpenSearch hits without catching validation failures. This catches ValidationError per hit and falls back to a stringified event, matching the existing fallback pattern in file_task_handler.py's _log_stream_to_parsed_log_stream. Related: apache#69304 * Lower malformed-log fallback logging to debug level _safe_build_structured_log_message runs once per stored hit inside the _read comprehension, so logging the fallback at warning level fires once per malformed line within a single log-fetch request. Drop it to debug so a request with many malformed entries does not flood the logs. * Set task state to SUCCESS in malformed-event read test The shared ti fixture defaults to RUNNING, and _read() now delegates running/deferred tasks to the base handler's remote-log path, which is not wired up in unit tests. Match the sibling read tests by setting the task to SUCCESS so the test exercises the stored-log fallback path.
4 participants
Summary
eventfield (for example, a task thatlogs a list or dict as the sole message argument) currently crashes the entire
task-log-fetch request with an unhandled pydantic.ValidationError, instead of
degrading gracefully.
_read()builtStructuredLogMessageobjects from stored OpenSearch hits withoutcatching validation failures. This change catches
ValidationErrorper hit and fallsback to a stringified event, matching the existing fallback pattern in
file_task_handler.py's_log_stream_to_parsed_log_stream. The identical pattern wasindependently fixed in the Elasticsearch provider in Prevent malformed Elasticsearch log entries from crashing task log fetch #69306.
Changes
providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py:wrap
StructuredLogMessageconstruction in a try/exceptValidationErrorwith astringified-event fallback and a warning log.
providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py: addcoverage for malformed
eventhandling and forread()surviving one bad hit amongotherwise-valid hits.
PR Checklist
mainbranchprekhooks all passRelated: #69304